1. Field of the Invention
The present invention relates to a method of searching or browsing data, and more particularly to a method of searching or browsing multimedia data such as an image or video.
2. Background of the Related Art
Recently, technology for digital image signal processing has been greatly developed and has been applied in various fields. For example, the digital image signal processing technology may be used in a search system for automatically editing only the face of a specific character in a moving picture file of a movie or drama, in a security system for permitting access to only persons registered in the system, or in a search system for searching a particular data from an image or video detected by a detection system. The performance of such systems basically depend on the accuracy and speed of detecting or searching the desired object. Accordingly, various image searching methods have been proposed in the related art.
An image search system which detects a degree of similarity with an image to be searched utilizing features such as color, texture or shape is disclosed in U.S. Pat. No. 5,579,471 entitled xe2x80x9cAn image query system and method.xe2x80x9d Depending upon the image to be searched, the importance of a feature may vary and within one particular feature such as the color, the importance of a feature element such as the red or green color may also vary. However, the above searching system does not take into consideration the different importance of features or feature elements for each image to be searched.
In another searching method entitled xe2x80x9cVirage image search enginexe2x80x9d (www.virage.com), a user directly inputs the level of importance for features such as the color, texture and shape by assigning weight values. Although an image may be searched according to an importance of a feature using this method, it may be difficult for a user to determine the importance of features.
Therefore, Yong Rui in xe2x80x9cRelevance feedback techniques in interactivexe2x80x9d SPIE Vol. 3312, discloses a method in which images similar to a reference image are found and the importance of features or weight for features are automatically obtained by calculating the similarities among the found images. However, the weight importance information is not maintained after a search for a specific image is finished and must be calculated for each image search, even for a same image.
In the image search and browsing system or the video (moving image) search and browsing system of the related art, information which describes a particular feature of an image or video data is utilized to perform a more effective search or browsing of the multimedia data. For example, in the image query system, an image may be divided into a plurality of regions and a representative color of each region may be utilized as a feature information of the image, or a whole color histogram of the image may be utilized as a feature information. Thereafter, two images are compared to calculate a similarity based upon the feature information and a determination is made whether the two images are similar.
Also, a user may wish to search and view a section of a video in which a specific character appears or a specific event occurs. At this time, a key frame representing a specific event may be utilized to perform a search and browsing of the video. Moreover, in recent searching and browsing methods, a key frame and a key segment have been defined utilizing an upper level feature information and a lower level feature information. This type of multimedia data search and browsing method may use weight information which indicates a degree of importance or may directly or indirectly use group information which was obtained in advance by grouping similar multimedia data. Such weight information or group information may be predefined in advance by an expert or may be a feature information variably modified by user feedback.
Furthermore, when browsing the multimedia data such as an image or video, the whole or a portion of the original multimedia data can be displayed depending upon a user request or upon a resource condition of the hardware. When portions of the original multimedia data are displayed, weight values are assigned to each portion such that an important portion among the whole data may be preferentially displayed. This weight value may be a varied by a user feedback or defined in advance by an expert.
In U.S. Pat. Nos. 5,020,019 and 5,297,042, information which describes a user preference is utilized to provide a better service for each user. For example, for a movie or video data, the information may include user preference such as action movie, drama or sports. In another example, users may have a different preference and meaning for keywords used most frequently in a keyword search. Therefore, each user can perform a more effective search if weights of keywords used often by a user is assigned differently according to the user preference. Such user preference may be obtained using search conditions and history of previous searches by the user. The information which depends on a user preference may also be varied and updated.
Although the variable feature information may be sufficient for some searching and browsing of multimedia data, when the feature information which includes the weight values is predefined by an expert, the reliability may vary depending upon the trustworthiness of an expert and/or a number of expert opinions used in defining the variable information. In other words, even if the feature information describing a feature of the multimedia data is defined in advance by an expert group, the reliability of the feature information defined for a same image may be inconsistent depending upon the skill of the expert. Similarly, when the feature information is trained by a user feedback, the reliability may vary depending upon the number and accuracy of feedbacks. For example, if two images are incorrectly judged to be similar by a user after a search, such information would be fed back and adversely affect the training of the feature information.
In addition, a user preference input by a new user may be different from the feature information recommended by a server. In such case, the system should assign appropriate weights to the user preference information depending upon the reliability of the user. Finally, as different users gives different feedback to the system, if the reliability of the existing feature information is relatively high, individual feedback of a new user would have a small effect on the update of the feature information. However, if the reliability is low, a feedback of each new user would have a large effect on the feature information value.
Since the searching and browsing methods of the related art do not take into consideration the factors described above, either the reliability of the searched result would be inconsistent or reliable feedback must be input at all times.
Accordingly, an object of the present invention is to solve at least the problems and disadvantages of the related art.
An object of the present invention is to provide an accurate and effective method of searching or browsing multimedia data.
Another object of the present invention is to provide a method of searching or browsing in consideration of a information reliability and an authority code to perform an update of the feature information.
A further object of the present invention is to provide a data structure for use in searching or browsing multimedia data.
Additional advantages, objects, and features of the invention will be set forth in part in the description which follows and in part will become apparent to those having ordinary skill in the art upon examination of the following or may be learned from practice of the invention. The objects and advantages of the invention may be realized and attained as particularly pointed out in the appended claims.
To achieve the objects and in accordance with the purposes of the invention, as embodied and broadly described herein, the method of searching or browsing multimedia data comprises selecting or inputting a reference multimedia data; searching for the reference multimedia data using weights assigned to features and/or feature elements of the multimedia data; inputting user feedback on the relevance of the searched or browsed multimedia data; measuring a similarity of the reference multimedia data to the resultant images of the search and calculating a new weight using a measured value; and updating previous weights and maintaining the updated weights.
In another aspect of the present invention, a method of searching or browsing multimedia data comprises searching a reference multimedia data using variable information representing importance of features of multimedia data, said variable information being included within the multimedia data structure, and using a reliability information representing a reliability of the variable information; inputting a user feedback on the relevance of the searched or browsed multimedia data; calculating a new variable information using training information fed back by a user or user pattern information stored in the system; and updating the previous variable information using the calculated variable information and/or reliability information, and maintaining the updated information.
In still another aspect of the present invention, a method of searching or browsing multimedia data comprises selecting of inputting a reference multimedia data; searching for the reference multimedia data using variable information representing importance of features of the multimedia data wherein the multimedia data structure includes the variable information, a reliability information of the variable information and an authority code for limiting an authority for a user; inputting a user feedback on a relevance of the searched or browsed multimedia data; measuring a similarity of the reference multimedia data to the resultant images of the search and calculating a new variable information using the measured value; and updating the previous variable information using a selective combination of the reliability information and authority code, and maintaining the updated information.
In still a further aspect of the present invention, a data structure for a multimedia data searching or browsing system comprises a multimedia data; feature information of the multimedia data; and weight information representing a weight of a feature of the multimedia data.